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PERFORMANCE  DIAGNOSIS  FOR  AIR  COMBAT  TRAINING 


Wayne  L.  Waag 
Armstrong  Laboratory 
United  States  Air  Force 
Williams  Air  Force  Base,  Arizona 


This  paper  summarizes  recent  research  at  the  Aircrew  Training  Research  Division  of  the 
Armstrong  Laboratory  that  is  attempting  to  enhance  air  combat  training  through  the  development  of 
performance  diagnosis  capabilities.  Three  systems  are  described:  the  Observing  System  for  Critique, 
Advice,  and  Review  (OSCAR),  a  system  which  provides  performance  diagnosis  capabilities  for  air 
intercept  training;  the  Air  Combat  Maneuvering  Performance  Measurement  System  (ACM  PMS),  a 
system  that  provides  real-time  measures  of  positional  advantage  and  energy  management  for  air 
combat  engagements  flown  in  either  a  simulation  or  range  environment;  and  the  Intelligent  System  for 
Air-to-Air  Combat  (ISAAC)  which  provides  diagnostic  feedback  capabilities  for  air  combat  maneuvering 
engagements. 


BACKGROUND  AND  RATIONALE 

Over  the  years  the  value  of  feedback  as  an  aid  to  learning  and  practice  has  been  well 
established.  In  fact,  the  finding  that  knowledge  of  results  leads  to  improved  performance  is  perhaps 
the  most  solidly  established  of  all  the  principles  of  learning.  However,  the  degree  of  information 
provided  by  feedback  in  most  real  world  settings  can  vary  dramatically.  Air  combat  training  is  certainly 
no  exception  and  can  be  characterized  by  the  development  of  a  variety  of  tools  to  enhance  the 
feedback  provided  in  both  simulated  and  real  world  engagements.  For  example,  gun  camera  film  has 
been  now  replaced  with  video  tape  that  can  provide  both  heads-up  display  and  radar  information. 
Instrumented  ranges  enable  the  accurate  positioning  of  aircraft  in  three-dimensional  space,  thus 
permitting  a  graphic  reconstruction  of  the  "big  picture"  on  an  air  engagement.  It  seems  to  be  fairly 
well-accepted  that  the  provision  of  such  engagement  feedback  is  of  benefit  to  training  for  air  combat. 

Despite  the  apparent  value  of  such  capabilities,  they  provide  only  a  reconstruction  of  the  events 
that  occurred  within  a  mission.  In  other  words,  such  systems  provide  a  very  good  reconstruction  of 
what  happened,  but  leave  the  diagnosis  of  whv  such  events  may  have  occurred  to  the  instructor  or 
observer.  Within  air  combat,  the  concern  for  training  is  not  so  much  the  final  outcome,  but  more 
importantly,  an  understanding  of  what  aircrew  actions  led  to  that  outcome.  For  that  reason,  research 
aimed  at  the  development  of  performance  diagnosis  capabilities  for  air  combat  training  was  initiated 
for  two  domains,  beyond  visual  range  (BVR)  air  intercept  training,  and  within  visual  range  (WVR)  air 
combat  maneuvering  (ACM).  At  present,  three  systems  have  been  developed  and  evaluated  in  the  lab. 
A  brief  description  of  these  three  systems  is  presented  below. 


OBSERVING  SYSTEM  FOR  CRITIQUE,  ADVICE.  AND  REVIEW  (OSCAR) 

In  1988,  a  contract  was  awarded  for  the  development  of  an  expert  system  enabling 
performance  diagnosis  for  basic  air  intercept  training.  This  effort  was  one  of  four  contracts  aimed  at 
the  application  of  artificial  intelligence  (Al)  techniques  (expert  systems  and  artificial  neural  networks) 
to  various  facets  of  air  combat  training.  The  goal  of  this  effort  was  to  develop  a  system,  OSCAR, 
which  could  provide  expert  critique  of  performance  as  training  feedback.  Once  developed,  an  attempt 
was  made  to  determine  whether  such  diagnostic  capabilities  had  utility  as  an  adjunct  to  basic  intercept 
training.  Only  a  very  brief  synopsis  of  this  research  will  be  presented  here.  A  complete  description 
of  the  effort  can  be  found  in  Obermayer  (1991). 
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An  overview  of  the  system  and  a  brief  description  of  its  capabilities  are  shown  in  Figure  1 .  As 
shown  OSCAR  takes  discrete,  time-sample  data  from  either  a  floppy  disc  or  direct  ethernet  interface 
from  the  host  system.  For  this  program,  the  Air  Intercept  Trainer  (AIT)  which  provides  basic  training 
for  either  the  F16A  or  F16C  aircraft  was  used  as  the  host.  Such  raw  data  are  reduced  into  a  file  of 
maneuver  events  and  measurements  which  could  be  noted  by  an  expert  observer.  Using  such  data, 
a  rule-based  expert  system  is  then  applied  which  draws  inferences  about  performance,  risk 
assessment,  and  probable  causes,  and  also  indicates  alternative  courses  of  action.  For  this  evaluation, 
these  inferences  were  based  upon  the  "textbook  solution"  as  found  within  current  tactics  manuals. 
These  inferences  are  then  integrated  with  a  dynamic  replay  of  the  intercept,  so  that  the  student  has 
information  not  only  on  the  actual  events  that  occurred  during  an  intercept,  but  also  a  critique  of  those 
events  and  a  statement  of  likely  behaviors  that  lead  to  such  observables. 

A  limited  evaluation  of  OSCAR  was  conducted  using  a  sample  of  1 1  instructor  pilots  at  Luke 
AFB,  Arizona.  The  evaluation  data  were  primarily  subjective  responses  to  a  set  questions  dealing  with 
the  perceived  value  of  OSCAR's  instructional  capabilities.  Overall,  the  results  were  encouraging 
indicating  that  OSCAR  was  easy  to  use,  and  would  have  a  significant  training  benefit,  especially  for 
individual  student  practice  and  self-debrief.  Such  data  suggested  that  such  performance  diagnosis 
capabilities  could  have  significant  value  for  both  simulation  and  range-based  systems. 


AIR  COMBAT  MANEUVERING  PERFORMANCE  MEASUREMENT  SYSTEM  (ACM  PMS) 

The  ACM  PMS  is  a  state  of  the  art  graphics  based  workstation  designed  to  "plug  in"  to  several 
major  aircrew  training  systems  to  support  both  operational  training  and  training  research.  The  system 
collects  data  in  digital  and  audio  form  and  provides  graphic  reconstruction  of  ACM  engagements  as 
well  as  a  variety  of  objective  performance  measures.  There  are  currently  three  operational  ACM  PMS 
systems,  two  of  these  are  installed  at  Luke  AFB,  and  one  at  Williams  AFB.  Of  the  two  systems  at  Luke 
AFB,  one  is  interfaced  with  the  Simulator  for  Air-to-Air  Combat  (SAAC)  and  the  other  with  the  Air 
Combat  Maneuvering  Instrumentation  (ACMI)  range  The  ACM  PMS  system  installed  at  Williams  AFB 
is  a  stand  alone  system  which  is  capable  of  replaying  engagements  recorded  on  either  of  the  two 
systems  located  at  Luke  AFB.  The  ACM  PMS  system  provides  the  following  general  capabilities  for 
the  SAAC  simulator  and  the  ACMI  range:  (1)  on-line,  real-time  data  sampling  during  ACM 
engagements;  (2)  on-line,  real  time  processing  of  sampled  data  with  presentation  of  selected  data  via 
CRT  display  for  immediate  feedback;  (3)  off-line  presentation  at  a  DEBRIEF  station  any  time  after  the 
engagement  of  the  same  data  as  above  together  with  cumulative  statistical  data  for  post-mission 
debrief  and  research  analysis. 

Each  ACM  PMS  consists  of  a  main  computer  suite,  a  display  system  and  ancillary  supporting 
computer  equipment.  At  present,  the  display  system  uses  Silicon  Graphics  Iris  equipment,  and  the 
computational  system  makes  use  of  Gould  Concept  Series  minicomputers.  A  common  interface 
between  each  system  is  accomplished  through  the  use  of  nine  track  magnetic  tape.  This  allows  the 
transfer  of  stored  SAAC  and  ACMI  engagements  to  be  replayed  and  analyzed  on  the  various  ACM  PMS 
systems.  The  system  is  currently  being  re-hosted  onto  a  Silicon  Graphics  Personal  Iris  workstation, 
which  will  dramatically  reduce  the  footprint  of  the  system  and  associated  maintenance  costs. 

As  stated  above,  the  ACM  PMS  acquires  sampled  data  in  both  digital  and  audio  form  from  the 
SAAC  and  ACMI.  This  data  is  available  to  the  user  in  several  forms.  A  graphic  reconstruction  of  an 
engagement  with  the  accompanying  engagement  audio  is  presented  through  the  graphic  display 
system.  This  graphic  reconstruction  is  available  in  both  real  time  and  for  post  mission  review.  In 
addition,  engagement  and  aircraft  parameters  in  alpha  numeric  form  are  available  through  the  database 
management  system  associated  with  the  ACM  PMS  for  post  mission  review  and  analysis. 
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OSCAR 

OBSERVING  SYSTEM  FOR  CRITIQUE,  ADVICE  AND  REVIEW 
FOR  AIR  INTERCEPT  TRAINING 


1_U  Hadar  setup  was  normal 
[~2l  Didn’t ADJUSTSPEED 

[31  Attacked  correct  TARGElj 

m  Shot#1  Scorel  =  0, 2=0 
Delay  .75 1  =  02=  0 


Theshotscoreis  a  function  of  the 
AntennaTrain  Angle.  Range,  and 
Aspect  Angfe.  The  Result  is  a 
rule-of-thumb  probability  of  kill 
at  the  time  of  the  shot.  The  delay 
score  is  the  amount  of  time  earlier 
that  a  .75  score  could  have  been 
achieved. 


1)  Nautical  miles  of  radar 
gate  and 

2)  +/-  degrees  of  radar 
azimuth  scan,  and 

3}  Number  of  bars  of  radar 
elevation  scan, 

then  it  Is  definite  (100%)  that 
Use  GCI,  situational 
awareness  or  briefing  to 
pointintherightdirection 
and  set  radar  at  the  dr  iefed 
range,  elevation,  sweep,  and 
scan. 


J  205 


I  SUN  386i  COLOR  GRAPHIC^  ‘  DEBRIEFING  DISP 


HOW  TIME-SAMPLED  AND 
ncpAD  DISCRETE  DATE  ARE 
WnS  COLLECTEDVIA 
WUrlKo  FLOPPY  DISK  OR 

DIRECTCONNECTION 


ASMARTWINDOW-BASED 
ASSESSMENT  AND  MEAS¬ 
UREMENT  INTERFACE  (SWAMI) 
CREATES  A  FILE  OF  MANEUVERS 
EVENTS  AND  MEASUREMENTS 
WHICH  WOULD  BE  NOTED  BY 
AN  EXPERTOBSERVER 


A  RULE-BASED  EXPERT 
SYSTEM  MAKES  INFERENCES 
ABOUTPERFORMANCE, 
REASSESSMENT,  PROBABLE 
CAUSES.AND  ALTERNATIVE 
COURSES  OF  ACTION 


PLAYBACK 

FEATURES 


o  CONTINUOUS  PLAYBACK  WITH  START/STOP  SET  BYTIME  OR  NARRATION  EVENT 
o  WHAT,  HOW,  AND  WHY  EVENTS  HAPPENED,  PLUS  ALT.  COURSES  OF  ACTION 
o  MULTIPLE  GRAPHIC  VIEWS:  B-SCOPE.TOP/SIDEVIEWS  WITH  VISIBLE  RADAR  BEAM 
o  AUTOMATIC  CRITIQUE  AS  PLAYBACK  PROCEEDS.  PLUS  HOW  CRITIQUE  FORMED 
o  STORED  DEMONSTRATION  PLAYBACKFILES 


Figure  1.  OSCAR  System  Overview  and  Capabilities 
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ACM  PMS  DISPLAYS 


The  graphic  reconstruction  of  an  engagement  includes  five  different  displays.  They  are;  Out  of 
Cockpit,  3  Dimensional  (God's-eye)  view,  All  Aspect  Maneuvering  Index  (AAMI)  and  Specific  Excess 
Energy  (P.)  time  history  curves,  Relative  Offensive  Maneuvering  Performance  (ROMP)  time  history 
curve  and  the  Energy  Management  display. 

The  out  of  cockpit  view,  which  can  be  seen  in  either  the  upper  right  or  upper  left  of  Figure  2, 
provides  a  computer  generated  view  of  the  engagement  as  seen  through  the  windscreen  of  the 
selected  aircraft.  This  view  covers  _+  60°  in  azimuth  and  -10°  to  +60°  in  elevation.  The  out  of  cockpit 
view  includes  some  aircraft  specific  Heads  Up  Display  (HUD)  like  information  such  as  the  airspeed  scale 
on  the  left  and  the  altitude  scale  on  the  right.  The  out  of  cockpit  display  also  provides  information 
relative  to  the  selected  pair  (2  Vs  1  bottom  center  of  the  display)  in  terms  of  Angle  Off  the  Tail,  Slant 
Range,  Antenna  Train  Angle  and  Closing  Velocity.  The  Out  of  Cockpit  Display  also  identifies  when  the 
aircraft  achieves  a  radar  lock  on. 

The  God's-eye  display,  which  is  shown  in  the  center  of  Figure  2,  provides  a  three  dimensional  view 
of  the  engagement.  The  viewing  angle,  azimuth  and  elevation,  and  the  distance  from  the  engagement 
may  be  manipulated  via  the  mouse.  Time  history  trails  of  aircraft  flight  path  are  also  available  on  the 
God's-eye  display.  The  length  of  these  trails  is  adjustable  or  can  be  eliminated  all  together.  The  God's- 
eye  display  also  contains  an  "altitude  pole"  which  displays  the  current  altitude  of  the  various  aircraft. 

The  AAMI/Ps  time  history  curves,  which  is  enlarged  in  the  center  of  Figure  3,  provide  a  graphic 
representation  of  the  AAMI  algorithm  (top  half  of  the  display)  for  a  selected  aircraft  pair  i.e.  1  Vs  2, 
and  a  graphic  representation  of  the  selected  aircraft's  energy  (bottom  half  of  the  display)  over  the 
length  of  the  engagement.  The  actual  display  is  in  color  and  AAMI  curves  are  computed  for  each  of 
the  available  weapons  and  are  displayed  simultaneously  in  color  coded  by  weapon  type.  This  display 
also  includes  a  shot  history  (top  of  the  display)  for  the  selected  aircraft.  A  more  detailed  discussion 
of  how  the  AAMI  is  developed  is  provided  in  the  Performance  Measures  section  below. 

The  ROMP  curve,  which  is  enlarged  in  the  center  of  Figure  4,  is  a  graphic  representation  of  the 
Relative  Offensive  Maneuvering  Performance  of  a  selected  pair  of  aircraft.  This  curve  is  generated  by 
subtracting  the  AAMI  of  the  second  aircraft  selected  from  the  AAMI  of  the  first  i.e.  Aircraft  #1  minus 
Aircraft  #2  as  in  Fig.  4,  with  the  first  aircraft  selected  (A/C  #1)  displayed  on  the  top.  This  curve 
shows  very  plainly  which  aircraft  had  the  advantage  at  various  times  throughout  the  engagement.  A 
more  detailed  discussion  of  the  ROMP  is  provided  in  the  Performance  Measures  section  below. 

The  Energy  Management  display,  which  is  shown  in  the  upper  right  part  of  Figure  5  and  also  called 
the  "Energy  Doghouse",  is  a  graphic  depiction  of  the  energy  states  of  a  selected  pair  of  aircraft.  The 
exterior  perimeter  of  the  "doghouse"  is  the  flight  envelope  of  a  given  aircraft,  color  coded  to  match 
the  A/C  number.  The  position  of  the  A/C  number  located  inside  the  doghouse  represents  the  current 
energy  state  of  that  aircraft.  The  line  which  nearly  bisects  the  interior  of  the  doghouse  horizontally 
represents  the  zero  P,  state  of  the  aircraft.  Flight  on  the  line  represents  sustained  performance.  Flight 
above  the  line  indicates  instantaneous  performance  or  energy  deficit  while  flight  below  the  line 
indicates  excess  energy.  A  more  detailed  discussion  of  Energy  Management  is  provided  in  the 
performance  measures  section  below. 
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Figure  2.  ACM  PMS  Display  Example 
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Figure  3.  A  AMI  and  Ps  Curve  Examples 
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Figure  4.  ROMP  Curve  Example 
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Figure  5.  Energy  Doghouse  Display  Example 


These  displays  are  presented  on  a  1 1  inch  by  1 4  inch  color  monitor  which  is  divided  into  six  display 
windows.  These  windows  are  upper  right,  upper  left,  center,  right  and  left  side  windows,  and  a  menu 
area.  Three  of  the  windows,  upper  left,  upper  right  and  center  are  considered  primary  and  can  be 
configured  with  any  of  five  displays  described  above.  Figure  2  shows  the  ACM  PMS  display  in  one 
of  the  "normal"  configurations,  i.e..  Out  of  Cockpit  #1  upper  left.  Out  of  Cockpit  #2  upper  right  and 
God's-eye  center.  These  six  display  windows  include  eight  functional  areas  (A  -  H)  as  described  below. 


Area 

Area  Name 

Area  DescriDtion 

A 

Upper  Left  Display 

This  is  a  7"  X  4"  area  which  depending  on  menu 
selections  displays:  Out  of  Cockpit,  God's-eye, 
AAMI  &  EM  Time  History  Curves,  and  ROMP. 

B 

Upper  Right  Display 

This  is  a  7"  X  4"  area  which  depending  on  menu 
selections  displays:  Out  of  Cockpit,  God's-eye, 
AAMI  &  EM  Time  History  Curves,  and  ROMP. 

C 

Center  Display 

This  is  a  10"  X  5.9"  area  which  depending  on 
menu  selections  displays:  Out  of  Cockpit,  God's- 
eye,  AAMI  &  EM  Time  History  Curves,  and  ROMP. 

D 

Left  Side  Window 

This  is  a  2"  X  5.9"  area  which  displays 
Maneuvering  Index  and  Energy  State  Scales,  Radar 
Lock  information  and  specified  A/C  parameters. 

E 

Right  Side  Window 

This  is  a  2”  X  5.9"  area  which  displays 
Maneuvering  Index  and  Energy  State  Scales,  Radar 
Lock  information  and  specified  A/C  parameters. 

F 

Menus 

This  is  a  14"  X  0.8"  area  which  displays  the  Menu 
Buttons,  used  to  select,  modify,  and  control  the 
screen. 

G 

Message  Area 

This  is  an  8"  X  0.3"  area  located  above  the  Center 
Display  area  (C)  which  displays  text  formatted 
event  messages  such  as  Shots  fired  and  Radar 
Locks. 

H 

Status  Line 

This  is  a  14"  X  0.3"  area  at  the  top  of  the  display 

reserved  for  status  information:  Exercise  Number, 
Engagement  number.  Run  Status  (Pause,  Slow 
Forward  etc.),  date  and  Elapsed  time. 


Primary  control  of  the  ACM  PMS  is  by  means  of  a  menu  system  which  is  displayed  on  the  very 
bottom  of  the  ACM  PMS  screen  (See  Area  F  Fig  2).  Access  to  these  menu  functions  is  provided 
through  a  touch  screen.  As  mentioned  previously,  control  of  the  viewing  angle  and  perspective  of  the 
God's-eye  display  is  provided  by  the  mouse. 
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ACM  PMS  PERFORMANCE  MEASURES 


Included  in  the  ACM  PMS  are  several  measures  designed  to  assess  ACM  performance.  These  are 
the  All  Aspect  Maneuvering  Index  (AAMI),  the  Relative  Offensive  Maneuvering  Performance  (ROMP) 
and  the  Energy  Maneuverability  assessment.  Detailed  descriptions  of  these  measures  may  be  found 
in  McGuinness,  Forbes,  and  Rhoads  (1984).  Validation  data  for  these  measures  can  be  found  in  Waag, 
Raspotnik,and  Leeds  (1992). 

All  Aspect  Maneuvering  Index 

The  AAMI  provides  a  measure  of  the  "effective"  proximity  or  positional  advantage  of  a  fighter 
relative  to  an  adversary  aircraft.  This  effective  proximity  is  defined  for  each  weapon  available  in  the 
fighter's  arsenal  at  any  specified  moment.  AAMI  values  range  from  zero  to  100,  where  zero  indicates 
no  offensive  position  and  a  score  of  100  indicates  an  optimal  offensive  position.  The  AAMI  is  based 
on  three  parameters  describing  the  inter-aircraft  geometry:  Antenna-train-angle  (ATA),  angle-off-tail 
(AOT)  and  range  which  are  defined  as  described  below: 

ATA  is  the  angle  between  the  longitudinal  axis  of  the  fighter  aircraft  and  the  radial  from  the 
fighter's  nose  to  the  adversary.  ATA  is  also  known  as  the  Line  of  Sight  Angle  and  is  expressed 
in  degrees  from  zero  to  1 80  where  ATA  =  0  "on  the  nose"  and  ATA  =  1 80  when  the  opponent 
is  directly  behind  the  fighter. 

AOT  is  the  fighter's  position  with  respect  to  the  tail  of  the  adversary  aircraft.  AOT  is  also 
expressed  in  degrees  from  zero  to  1 80  where  AOT  =  0  when  the  fighter  is  directly  behind  the 
opponent  and  AOT  =  180  when  the  fighter  is  "on  the  nose"  of  the  opponent. 

Range  is  the  distance  from  the  fighter  to  the  adversary  aircraft.  It  is  also  referred  to  as  slant 
range  (S/R)  and  is  measured  in  a  straight  line  from  the  fighter  to  the  target. 


The  AAMI  is  calculated  as  follows: 

AAMI  =  F(ATA)  X  WRM  (AOT,R) 

Where: 

F(ATA)  is  a  weighting  function  and  is  expressed  as 
F(ATA)  =  1 00[(90-ATA)/90]  for  ATA  <  or  =  to  90° 
or  F(ATA)  =  0  for  ATA  >  90° 

WRM  (AOT,R): 

In  theory,  weapon  envelopes  define  four  ranges: 

R1  -  Minimum  aerodynamic  range 
R2  -  Minimum  range  where  a  Pk  =  1 
R3  -  Maximum  range  where  a  Pk  =  1 
R4  -  Maximum  aerodynamic  range 
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Based  on  inter-aircraft  geometry,  AOT,  Range,  Vc  and  Altitude,  a  look  up  table  for  a  particular  weapon 
is  used  to  determine  where  in  the  envelope  the  fighter  is  located.  If  the  fighter  is  between  R2  and  R3 
the  WRM  (AOT,R)  function  is  assigned  1 .  If  the  fighter  is  between  R1  and  R2  or  between  R3  and  R4, 
an  interpolation  is  done  to  determine  the  value  of  WRM  (AOT,R).  For  example,  if  the  fighter  was  half 
way  between  R3  and  R4,  then  WRM  (AOT.R)  would  be  assigned  a  value  of  0.5.  A  problem,  however, 
that  has  affected  the  accuracy  of  the  AAMI  from  the  beginning  is  that  R2  and  R3  do  not  really  exist 
in  any  official  sense.  In  the  discussion  of  the  AAMI,  McGuinness,  Forbes  and  Rhoads  (1984)  the 
"weapons  range  model  serves  to  confine  the  AAMI  to  predefined  'range  windows'  for  the  respective 
weapons".  Further  "these  range  windows  correspond  roughly  to  the  maximum,  minimum,  and 
optimum  limits  of  the  firing  envelopes".  Unfortunately  there  is  no  discussion  of  the  derivation  of 
"optimum  limits".  As  a  result  the  current  AAMI  arbitrarily  uses  some  factor  times  R1  and  R4  to 
generate  R2  and  R3  respectively.  Although  this  has  been  shown  to  be  very  effective  in  identifying  the 
proper  trend  of  an  offensive  score  it  should  in  no  way  be  taken  as  an  absolute  offensive  score. 


Relative  Offensive  Maneuvering  Performance 

As  noted  above,  the  AAMI  reflects  the  offensiveness  of  an  aircraft  relative  to  it's  selected  pair. 
Since  AAMI  is  an  offensive  measure  only,  when  AAMI  goes  to  zero  it  is  impossible  to  determine  if  the 
aircraft  is  actually  neutral  or  defensive.  One  way  to  determine  when  one  of  the  participants  is  neutral 
or  defensive  would  be  to  look  at  the  AAMI  of  the  opponent,  or  what  is  referred  to  as  the  AAMI 
defensive  score.  The  ROMP  in  fact  does  this.  The  ROMP  is  a  composite  of  the  AAMI  scores  of  two 
aircraft  selected  as  a  pair.  It  has  a  value  of  -100  to  +100  and  is  derived  by  subtracting  the  AAMI  of 
one  aircraft  from  the  other.  For  example;  if  the  inter-aircraft  geometry  was  such  that  A/C  #1  had  an 
AAMI  score  of  100  and  A/C  #2's  AAMI  was  zero,  the  ROMP  for  #1  would  be  100  while  the  ROMP 
for  #2  would  be  -1 00  showing  that  #2  with  an  AAMI  of  zero  is  not  neutral  but  actually  defensive.  In 
a  situation  where  both  aircraft  had  equal  AAMIs,  the  ROMP  would  be  zero  showing  that  even  though 
both  had  positive  AAMI's  the  situation  was  actually  neutral.  This  information  is  then  plotted  over  the 
length  of  the  engagement  similar  to  the  plot  of  the  AAMI  except  that  zero  on  the  Y-axis  is  in  the  middle 
of  the  scale  and  it  goes  to  1 00  in  both  directions  therefore  graphically  showing  which  aircraft  had  the 
positional  advantage  at  any  given  time.  (See  Fig  4) 


Energy  Maneuverability 

Energy  Maneuverability  (EM)  indicates  how  well  a  pilot  utilizes  potential  and  kinetic  energy  in 
the  dynamic  environment  of  air  combat  maneuvering.  Energy  maneuverability  is  a  valuable  research 
tool,  offering  the  researcher  a  way  to  quantify  energy  management.  The  time  a  pilot  spends  above, 
on,  or  below  the  Ps  =  0  line  is  an  index  of  energy  management,  in  the  same  way  the  AAMI  is  an  index 
of  offensiveness.  Time  dependent  plots  of  the  fighter  and  the  adversary's  positions,  with  respect  to 
Ps  =  0,  are  an  indication  of  how  well  the  pilot  manages  energy  with  respect  to  the  adversary. 
Examination  of  EM  curves  provide  indications  of  trends  in  energy  use  juring  engagements  or  during 
specific  maneuvers.  The  energy  management  display  (Fig  5)  consists  of  a  maneuvering  triangle  or 
"doghouse"  defined  by  a  set  of  1 2  points.  Sets  of  these  points  are  stored  for  each  type  of  aircraft  at 
1 0,000  ft.  increments  from  Sea  Level  to  50,000  ft.  Indicated  airspeed  (knots)  is  presented  on  the  X- 
axis  and  turn  rate  (degrees/second)  is  on  the  Y-axis.  Curves  representing  lift  limit  and  structural  limit 
intersect  at  the  aircraft's  corner  velocity.  For  a  given  airspeed  the  aircraft  stalls  at  turn  rates  above 
the  lift  limit  and  incurs  possible  damage  at  turn  rates  above  the  structural  limit.  The  intersection  of  the 
lift  limit  curve  and  structural  limit  curve  is  known  as  corner  velocity  and  represents  the  instantaneous 
quickest  tightest  turn  possible  at  the  given  altitude. 
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INTELLIGENT  SYSTEM  FOR  AIR-TO-AIR  COMBAT  (ISAAC) 

In  an  attempt  to  further  develop  diagnostic  measures  of  air  combat  performance  and  to  better 
understand  the  underlying  cognitive  and  decision-making  processes  required,  a  project  was  initiated 
to  investigate  pilot  expertise  in  one-versus-one  (Ivl),  air-to-air  (A/A)  combat  (Thomas,  Obermayer, 
Raspotnik,  and  Waag,  1992).  The  objective  of  the  effort  was  to  develop  a  performance  model  of 
expert  pilot  behavior  in  air  combat.  The  model  would  serve  as  a  standard  for  air  combat  performance, 
and  actual  pilot  performance  that  deviated  from  that  standard  would  serve  as  data  for  diagnostic 
measures  of  pilot  performance.  Several  procedures  were  used  to  determine  which  would  be  most 
effective  for  future  knowledge  elicitation  work.  The  knowledge  base  elicited  by  these  methods  was 
then  incorporated  in  a  descriptive  model  of  expert  air  combat  performance.  After  additional  detailed 
information  was  obtained  from  a  subject-matter-expert  (SME),  the  knowledge  base  was  coded  in  the 
form  of  a  rule-based  expert  system. 


KNOWLEDGE  ELICITATION  METHODOLOGY 

The  subject-matter-expert  (SME)  used  in  this  investigation  is  a  former  F-4  pilot  with  combat 
experience.  He  was  director  of  the  SAAC,  and  responsible  for  its  operation,  maintenance,  and  upgrade 
to  the  current  F-15  and  F-16  configuration.  The  SME  has  flown  more  than  500  engagements  in  the 
SAAC  against  pilots  ranging  in  proficiency  from  novice  to  expert. 

The  SME  was  interviewed  to  obtain  the  top  level  goals  of  air  combat  maneuvering,  plus  the 
subgoals  and  operators  which  allow  the  pilot  to  achieve  those  goals.  Goals  include:  Place  the 
opponent  in  your  cone  of  fire  while  remaining  outside  your  adversary's  cone  of  fire.  Subgoals  include: 
Turn  faster  than  your  opponent  and  point  your  aircraft  in  the  direction  of  your  opponent.  Operators 
to  accomplish  these  goals  include:  Turn  in  the  direction  of  the  opponent,  place  him  on  the  plane  of 
the  vertical  stabilizer  (PVS),  and  pull  him  to  your  nose. 

The  SME  flew  a  series  of  Ivl  air  combat  engagements  against  an  another  former  fighter  pilot 
in  the  SAAC.  Each  pilot  began  the  engagements  in  either  an  offensive,  defensive,  or  neutral  position. 
Immediately  after  simulator  sessions  the  engagements  were  replayed  on  the  ACM  PMS.  The  SME  was 
video  taped  as  he  debriefed  his  own  performance  during  replays,  describing  his  goals,  actions  to  meet 
those  goals,  and  the  necessary  conditions  for  initiating  those  actions..  This  information  was  used  to 
confirm  data  collected  in  the  interviews,  to  elaborate  and  refine  the  goals  and  operators,  and  to 
determine  which  operators  were  appropriate  under  what  circumstances.  From  a  methodological 
standpoint,  this  procedure  appeared  appropriate  to  meet  the  aforementioned  objectives;  however,  it 
was  noted  that  the  SME  occasionally  confused  air  combat  engagements.  This  was  not  unexpected 
since  his  behavior  was  similar  in  several  engagements,  and  the  time  between  engagements  and 
debriefing  those  engagements  usually  exceeded  one  hour.  This  knowledge-elicitation  procedure  may 
be  appropriate  for  identifying  general  principles  of  air  combat  and  specific  instances  where  air  combat 
rules  apply,  but  may  not  be  ideal  for  investigating  specific  pilot  decisions  during  air  combat. 

Two  similar  knowledge  elicitation  procedures  were  then  used  to  add  more  air  combat  rules  to 
the  database.  The  SME  was  audio  taped  as  he  identified  his  air  combat  goals,  his  actions  to  achieve 
those  goals,  and  the  situational  variables  that  dictated  those  actions.  The  SME  again  flew  a  number 
of  engagements  against  the  same  former  fighter  pilot  in  the  SAAC  using  the  same  initial  conditions  as 
before.  In  one  condition,  the  SME  described  what  he  was  doing  and  why,  while  flying  in  the  SAAC, 
and  in  the  other  condition  he  did  so  immediately  after  flying  each  engagement.  These  methods 
resulted  the  identification  of  a  set  of  situational  assessment  variables  such  as  nose  position,  closure, 
energy,  aspect,  range,  and  altitude;  specification  of  additional  air  combat  rules;  and  identification  of 
more  instances  where  the  rules  apply.  It  was  noted  that  performing  in  the  SAAC  occasionally 
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interfered  with  the  SME's  ability  to  describe  and  interpret  his  actions.  In  the  other  condition,  where 
the  SME  debriefed  after  engagements,  he  occasionally  forgot  details  of  the  air  combat. 

A  final  procedure  was  used  to  refine  situational  assessment  variables,  to  elaborate  and  refine 
air  combat  rules,  and  to  identify  specific  instances  where  rules  apply  in  achieving  the  stated  goals  of 
air  combat.  The  SME  again  flew  a  number  of  air  combat  engagements  in  the  SAAC  against  the  same 
former  fighter  pilot.  In  this  condition,  opponents  did  not  describe  their  actions  until  the  end  of  each 
engagement.  Engagements  were  replayed  in  the  SAAC  so  that  opponents  could  see  each  engagement 
recreated  to  include  all  in-cockpit  displays  and  out-of-cockpit  visuals.  There  is  also  a  freeze  capability 
in  the  SAAC's  replay  function.  This  allowed  the  SME  to  freeze  the  replay  so  he  could  fully  elaborate 
on  what  he  had  done  and  why.  SME  comments  were  again  audio  taped.  As  would  be  expected,  this 
condition  placed  the  least  demands  on  the  SME's  memory,  and  with  the  absence  of  the  competing  task 
of  flying,  afforded  him  ample  time  to  describe  the  rationale  behind  his  air  combat  behaviors.  This 
procedure  will  be  used  in  the  future  to  elicit  knowledge  for  additional  SMEs. 

The  information  obtained  from  the  above  procedures  was  compiled  and  modeled,  and  then 
provided  to  the  SME  for  critique  and  review.  He  provided  additional  detail  to  existing  rules  and 
additional  air  combat  rules  that  were  not  derived  using  the  previously  described  methodologies.  The 
final  set  of  rules  were  then  prepared  for  inclusion  in  a  rule-based  production  system: 


PRODUCTION  SYSTEM  DEVELOPMENT 

The  purpose  of  production  system  development  was  to  provide  a  model  of  expert  performance 
in  F-1 5  and  F-1 6  air  combat.  A  requirement  of  the  expert  system  was  that  it  be  capable  of  interfacing 
with  the  SAAC  database  which  includes  500  engagements  between  our  SME  and  125  pilots.  The 
expert  system  was  to  include  a  performance  measurement  and  critiquing  capability,  and  be  easily 
validated  using  SAAC  engagement  data.  Validation  testing  would  include  determining  if  the  SME 
actually  followed  his  own  rules,  and  if  the  model  could  predict  air  combat  winners  and  losers,  and 
expert  and  novice  pilots. 

The  initial  expert  model  described  the  first  several  "moves”  appropriate  for  the  line  abreast 
initial  condition  where  opponents  start  at  co-altitude,  6000  feet  apart.  The  appropriate  things  to  do, 
according  to  the  SME,  are  to  turn  in  the  direction  of  the  adversary  until  the  opponent  aircraft  appears 
on  the  plane  of  the  vertical  stabilizer  (PVS)  (a  plane  that  extends  from  the  nose  to  the  tail  of  an 
aircraft),  and  then  pull  hard  on  the  stick,  so  as  to  point  the  aircraft  at  the  adversary.  Assuming  both 
opponents  do  the  same  thing  (if  one  does  not,  he  will  be  at  a  significant  disadvantage),  the  line  abreast 
condition  will  convert  to  a  near  head-on  condition.  If  one  of  the  opponents  achieves  a  significant 
aspect  angle  advantage  as  a  result  of  making  a  smaller  initial  turn,  that  pilot  should  reverse  the  bank 
of  his  aircraft  to  maintain  visual  contact  with  the  other  aircraft.  He  should  then  begin  a  lead  turn 
before  the  airplanes  pass  head-on.  This  set  of  actions  will  give  the  lead-turning  aircraft  an  additional 
aspect  angle  advantage  on  the  next  head-on  pass. 

The  above  example  was  coded  using  the  expert  system  shell,  PCPIus.  This  shell  requires 
production  rules  that  define  appropriate  aircraft  maneuvers,  additional  rules  to  define  these  maneuver 
rules  in  terms  of  aircraft  state  parameter  data,  rules  to  search  the  database  to  determine  if  state 
parameters  requirements  have  been  satisfied,  and  rules  that  report  if  aircraft  maneuvers  have  been 
performed  correctly.  The  resulting  expert  model  is;  Therefore,  a  hierarchy  of  rules  and  specifications 
to  include  goals,  top-level  rules,  translation  rules,  parameter  definitions,  special  default  provisions,  and 
software  to  process  engagement  data.  The  expert  system  is  primarily  a  backward  chaining  inference 
system,  that  proceeds  backward  from  specified  goals,  chaining  through  the  applicable  rules,  to 
determine  if  the  goal  was  satisfied.  Examples  of  goals  include:  a  maneuver  goal,  to  test  if  pilot 
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maneuvering  conforms  to  the  SME  rules,  and  goal  reporting,  to  ensure  that  output  required  for  future 
analyses  is  recorded. 

Top-level  rules  infer  a  yes/no  value  for  each  of  the  goals.  MAN  1  and  MAN  2  rules  are  each 
defined  in  terms  of  more  than  one  specific  pilot  action,  as  described  by  the  SME.  These  rules 
determine  if  Maneuver  1  and  Maneuver  2  were  performed  correctly. 

RULE  1 1 

Subject:  Maneuver  1  Rules 

If  :  (Roll  T oward_Bogey_Man  1  And  BogeyOn  PVS_Man1  And  Max 
_GJVIan1) 

Then:  (Turned_lnto_Bogey) 

Top-level  rules  use  terms  which  require  further  definition  to  determine  values  in  terms  of 
available  engagement  data.  For  example,  statements  such  as  "roll  toward  the  bogey"  and  "on  the 
PVS"  require  translation  rules  that  specify  the  behavior  that  defines  the  top-level  rules.  Defining  these 
translation  rules  often  required  additional  interaction  with  the  SME,  since  the  content  of  these  rules 
is  only  implied  by  top-level  rules. 

RULE  23 

Subject:  T ranslate-T o-Data-Man  1  -Rules 

If  :  ((Left_Abreast  And  Aroll_F_2  <  0  And  Aroll_F_4  <  0) 

or  (Right_Abreast  And  Aroll_F_2  >  0  And  Aroll_F_4  >0)) 

Then:  (Roll_Toward_Bogey_Man1) 

A  third  level  of  the  expert  system  is  parameter  definitions.  The  parameters  are  used  by  the 
translation  rules  and  must  be  defined  for  the  expert  system.  Parameters  are  defined  by  such  values 
as  number,  positive  number,  string,  and  single-  or  multi-valued.  Parameters  also  identify  data  from  the 
air  combat  engagement  file  by  file  name  and  location  in  the  file. 

AROLL  F  2 

Translation:  (Roll  Angle,  A/C  1,1/4  Betw  Start  And  CPA1 ) 

Type:  Singlevalued 

Used  By:  (Rule023) 

Method:  (Dos-File-ln  PCFILE.1 1  Index  58) 

Due  to  the  idiosyncrasies  of  PCEasy,  some  additional  information  must  be  provided  by  giving 
specific  values  to  parameters.  This  "special  case"  is  an  output  message  parameter  where  a  value  is 
the  message.  If  a  yes  value  for  a  yes/no  parameter  or  goal  cannot  be  established,  PCEasy  will  infer 
"don't  know”;  or  if  unable  to  establish  a  "yes"  value  should  result  in  a  "no"  value,  then  the  parameter 
must  be  given  a  "no"  default  value. 


MAN1 

Translation:  (Maneuver  Segment  1  Appropriate?) 
Type:  Yes/No 

Updated-By:  (RuleOlO  Rule055) 

Used-By:  (Rule033  Rule042  Rule009) 

Default:  (No) 
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Software  was  developed  to  process  air  combat  engagements  for  use  by  the  expert  system. 
A  "C"  program  was  developed  to  process  SAAC  engagement  files  into  a  form  suitable  for  the  expert 
system.  This  program  encodes  engagement  file  data  in  the  format  required  by  PCEasy;  parses  time 
history  files  into  segments  corresponding  to  commonly  occurring  events  such  as  start,  first  merge, 
second  merge,  and  various  typical  aircraft  maneuvers;  and  provides  computations  for  variables  such 
as  "on  the  PVS",  "bogey  in  view",  "percent  of  time  in  cone  of  fire",  and  percent  of  time  offensive, 
defensive,  or  neutral. 

Subsequent  work  involved  adding  air  combat  rules  to  the  existing  knowledge  base,  and  refining 
and  testing  those  rules,  and  partitioning  the  knowledge  base  into  frames  corresponding  to  commonly 
occurring  segments  of  air  combat.  The  first  frame  includes  rules  describing  initial  conditions,  rules 
required  to  bring  the  adversaries  to  the  first  merge,  and  rules  that  apply  at  and  immediately  following 
the  merge.  Since  air  combat  often  involves  a  series  of  near  head-on  passes,  the  second  frame  includes 
rules  that  describe  a  series  head-on  passes.  Frames  were  also  developed  to  include  rules  that  indicate 
what  the  pilot  should  do  when  he  is  either  offensive  or  defensive  with  respect  to  the  adversary.  A  final 
frame  was  developed  to  include  rules  that  apply  when  neither  pilot  is  offensive  and  both  aircraft  are 
in  close  proximity. 

Model  validation  is  currently  underway  and  will  determine  the  degree  to  which  the  performance 
of  the  model  compares  to  that  of  actual  pilots.  For  example,  can  the  model  predict  winner  versus  loser 
and  novice  versus  expert  performance,  and  does  the  model  compare  favorably  to  air  combat  rules 
specified  by  other  SMEs? 


FUTURE  PLANS 

At  present,  ISAAC  is  currently  under  evaluation  to  determine  the  validity  of  its  rule-base.  The 
initial  attempt  is  to  determine  whether  the  rules  that  have  been  developed  describe  the  actual  behavior 
of  the  subject-matter-expert  during  simulated  air  combat  engagements.  Once  the  validity  has  been 
established,  ISAAC  shall  be  integrated  with  the  ACM  PMS  to  permit  a  comprehensive  replay  and 
performance  diagnosis  debriefing  capability.  The  integrated  system  shall  be  evaluated  within  the 
context  of  air  combat  training  using  the  Simulator  for  Air-to-Air  Combat  (SAAC).  In  the  event  such 
capabilities  are  considered  to  be  of  value,  the  next  phase  of  the  program  would  entail  the  development 
of  such  diagnostic  capabilities  for  the  multi-ship  air  combat  arena.  -  Efforts  would  focus  on  both  the 
BVR  and  WVR  aspects  of  air  combat. 
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